Method and apparatus for the collection and management of quantitative data on unusual aerial phenomena via a citizen network of personal devices

ABSTRACT

The present invention relates to a method and apparatus to quantify unusual aerial phenomena via a network of citizen operated personal devices. A feature of the present invention is an app that runs on popular personal devices. A further feature of the invention is a server system in communication with said app. A further feature of the invention is a means by which a user having spotted a potential event can quickly engage the app to begin a data collection mode. A further feature of the invention is a data collection mode that simultaneously records data including but not limited to video, audio, 9-axis IMU data, GPS coordinates, and time. A further feature of the invention is a method of tagging the collected data with a cryptographic signature  800  to ensure integrity. A further feature of the invention is real-time app communication to a centralized server. A further feature of the invention is an alert to other users indicating something of interest is happening nearby. A further feature of the invention is the a means by which the server can distinguish interesting events from non-interesting events. A further feature of the inventions is a means by which the server can analyze the data to obtain scientifically useful quantitative information.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application is a utility patent. This application claims thebenefit of U.S. Provisional Application Ser. No. 63/326,014, filed onMar. 31, 2022 which is incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a method and apparatus for thecollection and management of quantitative data on unusual aerialphenomena via a citizen network of personal devices.

Description of the Related Art

Attempts have been made in the past to attain sighting information fromcitizens regarding unusual aerial phenomena. These include organizationsthat provide contact numbers whereby citizens can report theirobservations. Some organizations provide on-line forms to attempt togather relevant information, as well as to help process the volume ofincoming sighting reports. In some cases, personal device apps have beenprovided to assist in collecting citizen sighting information. Becauseof the nature and use of the sighting information, these data areconsidered anecdotal and not of scientific value. Some of these data maybe part of deliberate hoaxes.

What is needed is a method and apparatus whereby a network formed ofcitizens using personal devices can operate to collect data on unusualaerial phenomena that is of the quality and integrity sufficient to beconsidered scientifically quantitative and useful.

The present invention relates to a method and apparatus to quantifyunusual aerial phenomena via a network of citizen operated personaldevices.

A feature of the present invention is an app that runs on popularpersonal devices. A further feature of the invention is a server systemin communication with said app. A further feature of the invention is ameans by which a user having spotted a potential event can quicklyengage the app to begin a data collection mode. A further feature of theinvention is a data collection mode that simultaneously records dataincluding but not limited to video, audio, 9-axis IMU data, GPScoordinates, and time. A further feature of the invention is a method oftagging the collected data with a cryptographic signature to ensureintegrity. A further feature of the invention is real-time appcommunication to a centralized server. A further feature of theinvention is an alert to other users indicating something of interest ishappening nearby. A further feature of the invention is the a means bywhich the server can distinguish interesting events from non-interestingevents. A further feature of the inventions is a means by which theserver can analyze the data to obtain scientifically useful quantitativeinformation.

These and other features, aspects and advantages of the presentinvention will become better understood with reference to the followingdrawings, description and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the system configuration with a centralserver in communication with a network of personal devices.

FIG. 2 is a block diagram of a typical personal device comprisingsensors such as a GPS receiver, IMU sensor, precision clock, microphone,and camera, and controlled by a host processor in communication withlocal memory and an internet connection.

FIG. 3 is a state diagram showing typical modes comprising an idle modeand a recording mode, as well as transmission events info heartbeat,alert signal, and information packet.

FIG. 4 is a representation of the contents of the alert signal packetcomprised of a time stamp, an average camera pointing angle, an averageGPS position, a device ID, and a recording session ID.

FIG. 5 is a representation of the contents of the information packetcomprised of a start time, a stop time, an average camera pointingangle, a cryptographic signature, average GPS coordinates, a device ID,and a recording session ID.

FIG. 6 is a 3-dimensional isometric projection of the Cartesiancoordinate system with an equivalent image plane at distance f, andscreen coordinated unit vectors.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a method and apparatus for collecting andmanaging scientifically usable quantitative data on unusual aerialphenomena using a centralized server 110 and a network of citizens inpossession of personal devices 120. Personal device 120 contains sensors280 comprising a precision clock 210, a GPS receiver 220, a video camera230, a microphone 250, and a 9-axis IMU sensor 240. Sensors 280 ofpersonal device 120 are in communication with host processor 260. Hostprocessor 260 is in communication with internet connection 270 and localmemory 290. In the preferred embodiment 100, host processor 260 canaccess information from sensors 280, and/or control sensors 280. Furtherhost processor can transmit information through internet connect 270 toserver 110 and store information in non-volatile media storage 290.

In the preferred embodiment, a user downloads app 600 of the presentinvention on their personal device 120. App 600 gains permission toaccess elements 280. App 600 remains in idle mode 410 except to keeptrack of bias offsets and pointing direction of 9-axis IMU 240, and/orany other necessary housekeeping analytics or control as would be calledfor to enable the method of this invention. In the preferred embodiment,in idle mode 410, app 600 periodically sends its GPS position in an infoheartbeat 330 to server 110 for purposes as described below. If a userobserves an event of interest, the user can invoke a recording session500 of the app. In this mode the host processor 260 records data fromsensors 280 to local memory 290. In addition, host processor 260 underthe direction of app 600 sends relatively slow, periodic, repeatingalert signals 320 to server 110 through internet connection 270.

Alert signal 320 comprises a set of informative metrics regardingrecording session 500. These include but are not limited to the currenttime, average pointing angle, average GPS coordinates, and device ID.The server 110 receives this information transmitted from personaldevice 120 under the direction of app 600. The location information isused to check if other users in the network are nearby. In response toalert signal 320 from a personal device 120, and with general knowledgeof the location of all personal devices 120, server 110 then notifiesthe users of other personal devices 120, nearby to the positionindicated in said alert signal 320, that a possible unusual aerialphenomenon may have been spotted. The pointing angle and GPS location ofpersonal device 120 as indicated in alert signal 320 sent to server 100by direction of app 600 is interpreted by algorithms in server 110 andused to inform other users where they might find said object in the sky.The focus of recording session 500 is known to server 110 through alertsignal 320 by virtue of known GPS position and pointing angle ofpersonal device 120 as provided by GPS receiver 220 and 9-axis IMU 240.These data, by virtue of coordinate system 700. Origin 710 correspondsto GPS location and pointing angle corresponds to vector 740 extendingfrom origin 710 and extending in the direction 740. This implies apyramidal volume 720 extending to infinity within which the object ofinterest is expected to be contained.

In the preferred embodiment, if other users in the vicinity send alertsignals 320, the additional pointing angles obtained from their alertsignals 320 are used by algorithms in server 110 to further refine theobject location to better inform additional other users where to look.This occurs by virtue of the volume defined by the intersection ofpyramidal volume 720 of each personal device 120 involved in recordingthe same object.

When the user stops recording session 500, an information packet 310 issent by the method of this invention to server 110. Information packet310 comprises information that summarizes the data recorded by the userduring recording session 500 comprising a start time, stop time, averagepointing angle, average GPS coordinates, device ID, recording session500 ID, and a cryptographic signature 800. The cryptographic signature800 at minimum contains a hash of all the data that was recorded fromsensors 280. In the preferred embodiment, cryptographic signature 800relates to the hash of a recording data file 810 on storage 290comprising all the raw data recorded during recording session 500including all information required as per the method of this invention.Thus if any of the information of recording session 500 as stored inrecording data file 810 on storage 290 were to be altered, thecryptographic signature 800 hash would not match.

In the preferred embodiment, the cryptographic signature 800 is a hashof all the data recorded and stored in storage 290 as a single recordingdata file 810 during recording session 500. In an alternativeembodiment, the cryptographic signature 800 can be a crypto currencytransaction, or NFT (non-fungible token).

In the preferred embodiment, a record of information packet 310 is keptin storage 290. This serves as a means for users to confirm the findingsof the server 110 host. In one embodiment, an email containing thecryptographic signature 800 and other contents of information packet 310is sent to the email account of the user associated with the personaldevice 120 responsible for recording the associated recording session500. In this way, the data of recording session 500 stored in recordeddata file 810 is protected from attacks from the user side or the serverside.

In the preferred embodiment, after recording session 500, the user ispresented with an event form 330 that regards recording session 500.Event form 330 can be filled out partly, but requires a minimum set ofinformation that is necessary for the server to use to aid in vettingthe recording session 500 at the server. The user can access form 330 ofrecording session 500 later to fill in the remainder of the form thatbetter describes what the user witnessed.

In the preferred embodiment, subsequent to a data collection session500, an object detection neural net 900 is used on the local app 600 tohighlight detected objects. The user is prompted to assist the app 600in annotating the object. This helps in the case that other objects mayalso be present in the image but are known to the user to be not ofinterest.

In the event that a recording session 500 is invoked during a period inwhich personal device 120 is not in communication with the internet, theintegrity of recorded data file 810 does not benefit from stricttwo-sided attack protection. However, the method of this inventionincreases significantly the difficulty of a hoaxing attack involvingsynthetic data mimicking what is expected of the ensemble of sensors280.

In the event that multiple recording sessions 500 are captured for asighting of an anomalous object, by multiple devices 120 incommunication with server 110, it is assumed that the authenticity ofthe event is virtually guaranteed, as the creation of real-timesynthetic data required to hoax the event would be insurmountablydifficult by any standard.

In some cases, the object detector 900 may not be able to find theobject of interest in the images of recording session 500. In that case,the user is prompted to add in a bounding box to annotate the locationof the object in the images. Once the annotations of the video areacceptable to the user, the metadata describing the annotations are sentto the server.

In an alternative embodiment, in the case of multiple recording sessions500 having captured the same object from different directions, NeuralRadiance Fields (NeRF) or other techniques known to those skilled in theart may be employed to faithfully reconstruct animated views, and/or 3Dmodels of the object.

User calibration can be performed before, during, or after a sighting bypointing the camera at the sun or moon or constellation.

Server 110 may comprise a collection of servers located together orlocated apart and acting together in the manner described in the methodof this invention.

Server 110 must vet the data coming in from potentially millions ofpersonal devices 120 of the network of users. There are likely to bemany sightings of usual objects such as airplanes or stars. There willalso be reports associated with errors and tests and jokes and attemptedhoaxes. A feature of the present invention is the ability toautomatically vet recording sessions 500 to separate usual objects fromthe unusual objects that are the subject of this invention.

Recall that all data from recording sessions 500 is stored locally instorage 290 of users' personal device 120. Server 110 receives recordingsession 500 metadata from alert signals 320 and information packets 310.The data of recording session 500 may stay stored in recorded data file810 only in storage 290 on personal device 120 for a long period of time(days, weeks, months, years) without ever being uploaded to server 110.Only when a recording session 500 is deemed significant is all the datafrom recording session 500 as stored in recorded data file 810 uploadedfrom storage 290 of personal device 120 to server 110. In the preferredembodiment, the user is not involved in approving or being aware of theupload of recorded data file 810 of their recording session 500.

In an alternative embodiment, recorded data file 810 may be uploaded toserver 110 and/or deleted from personal device 120 storage 290 after apredetermined amount of time in order to better manage storage 290 onpersonal device 120. In this embodiment it is somewhat more likely thatimportant information could be lost by deletion before a determinationof its importance is made.

In the preferred embodiment, information packets 310 for all recordingsessions 500 are saved at the server 110 in a database. The amount ofdata per information packet 310 is fewer that 100 bytes, so that evenone billion information packets 310 could easily be stored. Further,they are each time-tagged, so that they can be stored by date andarchived. In the preferred embodiment data is kept indefinitely to allowfor retrieval at some time in the future. In an alternative embodiment,data may be kept only a maximum time before being deleted in order tobetter manage storage space on server 110.

At the first level, the required information of event form 330 is usedto determine if recording session 500 is of significance. The lack of asubmitted event form 330 is the first level of disqualification. Eventform 330 collects other information from questions targeted directly atthe nature of the event to further disqualify recording session 500. Anexample question may be, “Was this event a mistake?” Or, “Was this asighting of something of interest?”

In the preferred embodiment, recording sessions 500 are disqualifiedunless multiple users record the same event. The average GPS locationand average pointing angle metadata available from alert signal 320 andinformation packet 310 are used by the server to determine whethermultiple users could have recorded the same object. If no such multiplesighting exists for a recording session 500, it is disqualified forfurther study until and unless further information is revealed in thefuture that warrants further investigation. At that time, the data ofrecording session 500 as stored in recorded data file 810 can beuploaded from the users personal device 120 to server 110.

In the preferred embodiment, the raw data of the 9-axis IMU is recordedin real-time during recording session 500 into recorded data file 810 onstorage 290. These raw data are in integer counts of the IMU signals,and contain unwanted biases that are normally removed during processing.In the preferred embodiment, these biases can be removed in a two-passor Weiner algorithm known to those skilled in the art. In the two-passalgorithm, the IMU data are processed by means well known to thoseskilled in the art, such as by the method of Magwick, known to thoseskilled in the art. When the full data stream has been processed, thefinal biases can be used to initialize the biases for a second pass.Further, GPS position can be used to place an upper bound onaccelerations. These approaches amount to estimation by the technique ofWeiner rather than Kalman, and can be used to estimate absolute positionand relative orientation of the camera with relatively high accuracy.The techniques by which this is accomplished are known by those skilledin the art.

By this method, a high-quality estimate of camera absolute position andrelative pointing angle can be determined. And with this information,the vector 750 that describes the direction pointed to by any pixel inthe recording can be determined, subject to knowledge of a focal planeparameter 740 f. With the computed rotation matrix R we find the vectordefined by a given s_(x), s_(y) pixel coordinate as

$\begin{matrix}{{\overset{¯}{v}}_{a} = {{{fR}\begin{bmatrix}0 \\1 \\0\end{bmatrix}} + {s_{x}\hat{u}} + {s_{y}\overset{\hat{}}{v}}}} & (1)\end{matrix}$ where $\begin{matrix}{\hat{u} = {R\begin{bmatrix}1 \\0 \\0\end{bmatrix}}} & (2)\end{matrix}$ $\overset{\hat{}}{v} = {R\begin{bmatrix}0 \\0 \\1\end{bmatrix}}$

And f is the equivalent focal length to the image plane 730. In the caseof fixed lens, the focal plane parameter 740 f is simply a parameter ofthe model of personal device 120. In the case of a variable zoom lens,the focal plane parameter 740 f can be determined by personal device 120camera model 820, and with the camera metadata that describes the camerazoom factor. A lookup table of camera model 820 or curve is used torelate the camera zoom factor of the metatdata stored in recorded datafile 810 to the focal plane parameter 740 f.

A vector pointing in the direction of an object is determined based onthe xy coordinates of said object in camera 230 image and the pointingangle of camera 230 of personal device 120. If two or more recordingfiles 810 from two or more personal devices 120 exist regarding the sameevent, and if the two or more said personal devices are separated inlocation sufficiently to form an adequate baseline, triangulation can beused to determine the position of said object.

In general a plurality of vectors describing the pointing angle to anobject will violate epipolar constraints due to inaccuracies. For thisreason the preferred embodiment computes the location of the objectusing least-squares triangulation wherein a position is found such thatthe sum of the pointing errors of all contributing vectors is minimizedso as to best estimate the actual location of the object of interest.

The degree to which the epipolar constraint is violated serves as ametric on the accuracy of the camera 230 pointing computation. Thus,error bars can be determined by those skilled in the art as is necessaryin the pursuit of scientific quality data as per the intent of thepresent invention.

With 9-axis IMU 240 computations the relative orientation estimation ishighly accurate up to a fixed rotation dependent upon magnetic compassbiases, disturbances, and inaccuracies. Thus, if the data from tworecording sessions 500 from two users at two different locations areused to triangulate position, we can expect errors. In the preferredembodiment, a more accurate absolute pointing angle can be determined bycorrelating landmarks in the image with landmarks at the site where therecording session 500 was recorded. This requires a visit to the site,and is reserved only for cases that, if confirmed, would justify theeffort. In some cases, this refinement of absolute pointing angle couldbe performed without visiting the site provided distant landmarks can beseen prominently.

A usual problem with modern personal device 120 cameras 230 is that theyare not designed to focus on tiny objects in a uniform background, suchas would often be encountered in the hunt for unusual aerial phenomenon.In the preferred embodiment, the focus is set for infinity when therecording session 500 is instantiated. This is because the expectationis that the object being observed is greater than 100 meters away fromthe camera, so that infinite focus is ideal.

In an alternative embodiment, the app includes specialized algorithms todetect the object within the frame and measure its response to focuscommands. In that way focus can be adjusted in order to minimize and/orclarify the object of interest.

In either focus adjustment embodiment of the current invention, buttonsdisplayed prominently on the screen allow the user to exit the recordingsession 500 default focus mode and either choose a manual focus mode orallow personal device 120 to take control of focus adjustments.

The embodiments of the invention described herein are exemplary andnumerous modifications, variations and rearrangements can be readilyenvisioned to achieve substantially equivalent results, all of which areintended to be embraced within the spirit and scope of the invention asdefined in the appended claims.

1. Apparatus comprising a means to simultaneously record data fromseveral sensors;
 2. The apparatus of claim 1 wherein the sensorscomprise a video camera;
 3. The apparatus of claim 2 wherein the sensorscomprise a microphone;
 4. The apparatus of claim 3 wherein the sensorscomprise a GPS receiver;
 5. The apparatus of claim 4 wherein the sensorscomprise an inertial measurement unit;
 6. The apparatus of claim 5wherein the sensors comprise a precise clock;
 7. The apparatus of claim6 wherein data from the sensors is stored to local device storage inreal time on command of the user.
 8. Method whereby personal devices arein communication with a server, those devices sending metadata to saidserver describing recording sessions involving said several sensors; 9.The method of claim 8 wherein said metadata provides sufficient sensorinformation to locate device position and pointing angle in 3D space;10. The method of claim 9 wherein said server determines the validity ofsaid recording sessions by determining if multiple said sessions relatedtogether in 3D space and at the same time;
 11. The method of claim 9wherein said personal devices provide a periodic heartbeat comprising 3Dposition to said server;
 12. The method of claim 11 wherein server,being aware of 3D position of said personal devices, alerts the users ofnearby personal devices in the case one or more personal devices isengaged in a recording session;
 13. Method of claim 8 wherein a databaseof metadata is kept whereby said server can use said metadata todetermine the validity of said recording sessions;
 14. The method ofclaim 8 wherein in which devices and some of the sensors can becontrolled;
 15. The apparatus of claim 8 wherein the metadata includes ahash of the local file containing recorded data from said recordingsession;
 16. The method of claim 15 wherein said hash is sent via emailto the user associated with said personal device responsible forinvoking said recording session;
 17. The apparatus of claim 7 whereinsaid personal device controls operating parameters of said sensors,comprising, but not limited to, control of the camera focus, cameraexposure, camera zoom, and microphone gain;
 18. The apparatus of claim17 wherein said personal device controls said sensors to optimize therecording of anomalous aerial phenomena in daytime or nighttime; 19.Method wherein a server in communication with a plurality of personaldevices, organizes and orchestrates the coordination and collection ofmultiple sensor data so as to provide quantitative data of scientificquality on anomalous objects;
 20. Apparatus comprising softwareoperating on a personal device wherein the collection of quantitativedata of scientific quality on anomalous objects is made possible throughthe careful management of a plurality of unskilled users forming anetwork;
 21. Method wherein an combination of metadata packets andinstruction messages related to a large number of sightings of objectsby a network comprising a large number of users can be automaticallyprocessed in an efficient and accurate manner so as to make said networkfeasible and productive for the study of anomalous objects;